Core: Add freshness-aware loading to RESTTableOperations#16319
Core: Add freshness-aware loading to RESTTableOperations#16319yadavay-amzn wants to merge 1 commit into
Conversation
When RESTTableOperations.refresh() receives an ETag from the server, subsequent refresh() calls send If-None-Match and skip metadata re-download on 304 Not Modified. This reduces bandwidth and server load for repeated refresh calls (retries, transactions, reconciliation). Closes apache#15109
yadavay-amzn
force-pushed
the
fix/15109-rest-table-ops-freshness
branch
from
b2a2850 to
b585f22
Compare
|
@gaborkaszab This implements the freshness-aware loading follow-up you suggested in #15109. CI is green. Would appreciate a look when you get a chance. |
|
Thank you for the PR @yadavay-amzn ! |
|
@gaborkaszab Thanks for the review! Good question on the two ETags. The cache-level ETag (in They serve different purposes:
They can diverge temporarily (e.g., after a commit updates the ops ETag but before the cache is invalidated), but this is safe because both are optimistic -- a stale ETag just means an extra round-trip, never stale data. That said, if you feel strongly about unifying them, I could propagate the ops-level ETag back to the cache on refresh. Would that address your concern? |
2 participants
Summary
Adds ETag/If-None-Match support to
RESTTableOperations.refresh()andcommit(), so repeated refresh calls skip re-downloading unchanged table metadata.Follow-up to #15109 (freshness-aware table loading in REST catalog).
Changes
When
RESTTableOperations.refresh()receives an ETag from the server, subsequentrefresh()calls sendIf-None-Matchand return the current metadata on304 Not Modified. The ETag is also captured from commit responses so the next refresh benefits immediately.This reduces bandwidth and server load for repeated refresh calls (retries, transactions, reconciliation) without any API changes or cache coupling.
Scope
This is the minimal self-contained slice of #15109. Deliberately out of scope:
finalfield, needs design discussion)Testing
5 new tests in
TestRESTTableOperationsFreshness: